Good-Toulmin type estimators for the number of unseen species

Many scientific disciplines use n samples to extrapolate information on the species composition of additional unobservable samples. Given a population with unknown species proportions (pi)i≥1, we consider the problem of estimating: i) the number Uλn,r of unseen species with frequency r ≥ 1 in an additional sample of size λn; ii) the probability Vλn,r of discovering at the (λn + n + 1)-th draw a species with frequency r ≥ 0 in the enlarged sample of size n + λn. The former is related to the problem of controlling how many unseen species are rare, whereas the letter is related to the problem of evaluating the cost-effectiveness of further sampling. We introduce nonparametric empirical Bayes estimators of Uλn,r and Vλn,r, we show that they estimate Uλn,r and Vλn,r all of the way up λ ∝ log2(n)/2r log2(log2(n)), and that this range is the best possible. The proposed estimators are distribution-free, namely no assumptions are imposed on (pi)i≥1. We then consider tuning our estimators for heavy-tailed (pi)i≥1. To do that we impose regular variation as a distribution-specific assumption for the pi’s. Interestingly, the resulting regularly varying estimators are asymptotic equivalent, for large n, to their Bayesian nonparametric counterparts under a Poisson-Dirichlet prior for (pi)i≥1.